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ABSTRACT 

Image gathering and coding are commonly treated as tasks separate from each other and from 
the digital processing used to restore and enhance the images. Our goal in this paper is to develop 
a method that allows us to assess quantitatively the combined performance of image gathering and 
coding for the digital restoration of images with high visual quality. Digital restoration is often 
interactive because visual quality depends on perceptual rather than mathematical considerations, 
and these considerations vary with the target, the application, and the observer. Our approach is 
based on the theoretical treatment of image gathering as a communication ch ann el [J. Opt. Soc. Am. 
A2, 1644 (1985); 5,285 (1988)]. Initial results suggest that the practical upper limit of the information 
contained in the acquired image data ranges typically from ~ 2 to 4 binary information units (bifs) 
per sample, depending on the design of the image-gathering system. The associated information 
efficiency of the transmitted data (i.e., the ratio of information over data) ranges typically from 
~ 0.3 to 0.5 bif per bit without coding to ~ 0.5 to 0.9 bif per bit with lossless predictive compression 
and Huffman coding. These upper limits of performance are reached when the sampling passband 
of the image-gathering system closely matches the Wiener spectrum of the incident radiance field. 
The visual quality that can be attained with interactive image restoration improves perceptibly as 
the available information increases to ~ 3 bifs per sample. However, the perceptual improvements 
that can be attained with further increases in information are very subtle and depend on the target 
and the desired enhancement. 


1. INTRODUCTION 

Image gathering and coding are commonly treated as tasks separate from each other and from 
the digital processing used to restore and enhance the images. Ordinarily, image-gathering systems 
are designed to produce good visual quality for conventional image displays, and data-compression 
techniques are developed to reduce, as much as possible, the data necessary to reproduce a faithful 
duplicate of this original image. Image restoration and enhancement, despite the rapidly increasing 
use of digital processing, are virtually ignored in these assessments of image gathering and coding. 

Digital image restoration is often interactive because a single figure of merit for visual quality 
does not exist to formulate a single “best” algorithm. Visual quality is too elusive a concept for 
such a figure to exist. It depends on a number of attributes, such as fidelity (resemblance to the 
scene), resolution (minimum discernible detail), sharpness (contrast between large areas), and claxity 
(absence of visual artifacts and noise). The trade-off between these attributes of image quality still 
must be based on perceptual rather than mathematical considerations, and these considerations 
vary with the target, the application, and the observer. In addition, sometimes the enhancement of 
certain target features is desirable to improve resolution and sharpness, even at the cost of fidelity 
and clarity. 
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In previous papers Huck et al . 1,2 have shown that image gathering can be treated like a 
communication channel if (and only if) the image-gathering degradations are correctly accounted for 
in image processing. If this is done, then the informationally optimized image-gathering system tends 
to maximize the fidelity and robustness of a variety of optimally restored representations ranging 
from images to edges. It also is possible with interactive image restoration to improve significantly 
on the visual quality produced by the traditional methods employed in digital image gathering 
and restoration . 3,4 These traditional methods 5 ' 10 often have failed to improve on the visual quality 
obtained in a simpler and faster way by image reconstruction and interpolation. It is perhaps for 
this reason, at least in part, that image gathering and coding have not been assessed directly with 
digital restoration and enhancement in the past. 

In this paper we extend the information theoretic assessment of image gathering to include 
image coding. Our goal is to develop a method that allows us to assess quantitatively the combined 
performance of image gathering and coding for the interactive restoration of images with high visual 
quality. The major questions we deal with are: How much visual information can be acquired by 
the image-gathering process? How much information is required to restore images with high visual 
quality? And how can this information be transmitted most efficiently? We do not attempt here to 
compare the performance of a variety of image-coding techniques. Instead, we limit our assessment 
to the familiar lossless predictive compression with Huffman coding. 
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2. OUTLINE 


Figure 1 presents the end-to-end block diagram of the image gathering, coding, and restoration 
processes that we analyze in this paper. Our approach is to assess quantitatively the flow of 
information through the image gathering and coding processes, followed by a qualitative assessment 
of the visual quality that can be restored from the transmitted information. 
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Figure 1. Model of image gathering, coding, and restoration. 

In Section 3 we assess the information density of the data that is acquired by the image-gathering 
process in terms of the Wiener spectrum of the radiance field, the design of the image- gathering 
system, and the dynamic range and quantization intervals of the quantizer. We also introduce the 
concept of information efficiency (i.e. , the ratio of information over data) as an additional criterion 
of the effectiveness of image gathering and coding. Our formulations are based on the theoretical 
treatment of image gathering given by Iluck et al. 1,2 Following the methods of Shannon 11 and of 
Fellgett and Linfoot, 12 this treatment is constrained by the assumption that the radiance field and 
the noise are wide-sense-stationary Gaussian random processes. 

In Section 4 we assess the effects of image coding on the information efficiency of the transmitted 
data. We use the familiar lossless predictive compression together with Huffman coding. The 
predictive compression reduces the statistical redundancy of the digital data without loss of 
information, and the Huffman coding compresses the data by transmitting the more probable symbols 
in fewer bits than the less probable ones. In addition, we demonstrate that important differences 
exist between the information density of the transmitted data and the entropy that is often used in 
the prevailing digital processing literature 5 " 10 to assess data compression. 

In Section 5 we assess the quality of the restored images as a function of the available information. 
We first consider fidelity-maximized restorations. These restorations allow us to perform parametric 
trade-offs in terms of a single figure of merit, namely, the image fidelity. However, the fidelity- 
maximized images exhibit some visual defects such as ringing, aliasing artifacts, and noise. Thus, we 
use the Wiener-Gaussian enhancement (WIGE) filter introduced by McCormick et al. 4 to suppress 
these defects and improve the visual quality. 
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3. IMAGE GATHERING 


A. Information Capacity 

Let the image-gathering system acquire information about some isoplanatism area A of the 
radiance field L(x,y ) with the average power <r£. Furthermore, let the image-gathering process 

be constrained, like a communication channel, only by the frequency passband B and the white 
noise n(x,y ) with the power cby. Then the absolute upper limit of the acquired information about 
the area A is defined by the expression 

H c = \\A\\B log 2 [l + (K(T L /a N ) 2 j , 

where Ka^/cr^ is the rms signal-to-noise ratio (SNR) and K is the steady-state gain of the radiance- 
to-signal conversion in the image-gathering process. The associated information capacity of the 
image-gathering process per unit area A and unit passband B is 

h c = 77-7*7 - \ lo §2 [l + { Ko L / a N ?\ . (1) 

The magnitude of h c may be defined as bifs, binary information units per unit area and passband, 
analogous to bits for the binary units per sample of the transmitted digital data. 

The information capacity h c is plotted as a function of the SNR Kcr^/a^ in Fig. 2. It varies from 
~ 1 bif for A'cr^/cr/v = 2 to ~ 10 bifs for Ka^/a^ = 1000. This is the range of SNR’s that is typically 
of interest. SNR’s below this range ordinarily do not permit the restoration of images with good 
visual quality, and SNR’s above this range ordinarily do not improve the visual quality. In practice, 
of course, information is inevitably lost because the Wiener spectrum of random radiance fields is 
not white and band-limited to B and because the image-gathering process introduces aliasing and 
blurring as well as noise. In addition, information ordinarily is lost by the signal quantization that 
is required for digital data transmission and processing. 
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Figure 2. Information capacity h c versus SNR Kai/cr^. The image-gathering process is assumed to be constrained, 
like a communication channel, only by the sampling passband B and the SNR KcTi/a^. The radiance field 
spectrum is assumed to be white and band limited to B. 
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B. Radiance-Field Properties 


We assume that the incident radiance field L(x,y) consists of contiguous rectangles whose 
sides are parallel to some axes ( x',y ') (see Fig. 3). The transitions along each axis obey the 
Poisson probability-density function with the (expected) mean separation A -1 , and the radiance- 
field magnitude of each rectangle obeys the zero-mean Gaussian probability-density function with 
the (expected) variance a 2 y The resultant autocorrelation of L(x,y) is 13 

®l(x, y) = °L ex P “ c ) (M + b'Q] 

= ^i ex p [- (l*'l + ly'l) hA 

where c is the correlation of the radiance-field magnitudes of adjacent rectangles and, for convenience, 
p = 1/A(1 — c). If we let the orientation of the ( x ', y ') axes of the rectangles be random with uniform 
probability, then the autocorrelation becomes circularly symmetric as given by 

= °i ex p(-M/p)> ( 2a ) 

where r 2 = x 2 + y 2 . The corresponding Wiener spectrum of L(x,y ) can be closely approximated 
by l-4, 14,15 

-V 27T/i 2 (T? 

«F l (u, w) = <b L (p) = ( 2b ) 

[l + (2 w) 2 ] 3/ 

where p 2 = v 2 + w 2 . Figure 4 illustrates the normalized auto correlation <F^(a :,y) — ct£ 2< I>£ / (x, y) and 

the normalized Wiener spectrum $^(u, w) = (t^ 2 ^l(v, w). The mean spatial detail of the radiance 

field is conveniently represented by p. This implies that the correlation is c « 0.3. The exact 

expression for the Wiener spectrum of the target shown in Fig. 3 is given by Fales et al. 3 [Eq. (18)]. 
The Wiener-spectrum curves for that expression are almost identical to those given for the more 
convenient Eq. (2b). 




(a) Target 


(b) Sampling lattice 


Figure 3. Random radiance field with mean spatial detail y = 3, and the sampling lattice with unit sampling intervals. 
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Figure 4. Autocorrelation functions and Wiener spectra of the radiance field for several mean spatial details /x. 

C. Image- Gathering Degradations 

Conventional image-gathering systems consist of an objective lens (or lens system) and some 
sort of photon-detection and sampling mechanism. The most common mechanisms are sensor-array 
and line-scan devices. The lens and photosensor apertures are basically low-pass spatial-frequency 
filters. The spatial-frequency response of the image-gathering system, which is the product of these 
two low-pass-filter responses, ordinarily decreases smoothly with increasing spatial frequency until 
the lens diffraction limit is reached. 

Figure 5 presents a model of the image-gathering process that transforms the continuous radiance 
field L(x,y ) into the signal s g (x, y ) as defined by the expression 


s a (x,y)= [KL(x,y)*T g (x,y)]\\\(x,y) + n(x,y), (3a) 

where K is the steady-state gain of the (linear) radiance-to-signal conversion, n(x, y) is the (additive, 
discrete) sensor noise, * denotes convolution, and |||(x,j/) denotes sampling. Taking the Fourier 
transform of s g (x, y) yields the spatial-frequency representation of the acquired signal 

Sg(v,w) = KL(v,w)fg(v,w) * |]|(i;,it>) + n(u,tc), (3b) 

where L(v,w) and h(v,w) are the spatial-radiance and noise transforms, respectively, and r g (v,w) 
is the spatial-frequency response of the image-gathering system. The sampling function is given by 


OO CO 


jjj(u, w) = 6 ( v ~ m > w ~ n ) 


771— — CO 71= — OO 


= tf(W.U>)+ _£ Q 0 ( V ’ U,) 


for unit sampling intervals. The term — (u.rc) accounts for the sampling sidebands. 

T~ U> vJ 
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Figure 5. Model of the image-gathering process. 


It is convenient to normalize the sampling intervals to unity, and to define the frequency passband 
B as the sampling passband given by 


B = {(w, w), |u| < 0.5, |m| < 0.5} . (4) 

The corresponding area in the frequency domain is |R| = 1. The low-pass frequency response of 
conventional image-gathering systems can often be approximated by the Gaussian form 16 


r g (v, w) = exp 


~(p/ P c ) 2 , 


( 5 ) 


where the optical-design parameter p c is the spatial frequency at which r g (v,w) — 1/e = 0.37. 

If we now let the image-gathering process be constrained by the response r g (v,w), the sampling 
passband B, and the SNR Kop/oy, then the information density h g of the acquired signal s g (x, y) 
becomes 1 


h g — ~ J J 1°S2 

B 

The associated variance of the signal is 

oo 

<?g = JJ w )\ f g( v > w )\ 2 dv dw • ( 7 ) 

— OO 

Figures 6 and 7 illustrate the dependence of the information density h g on the optical-design 
parameter p c for several mean spatial details p and SNR’s Kai/aj\f. These results suggest the two 
following generalizations: 

(1) The information density h g tends to be maximum when the sampling passband most 
closely matches the radiance-field spectrum, regardless of the design of the image-gathering 
system. This occurs, for the target characterized by Eqs. (2), when the sampling intervals 
are approximately equal to the mean spatial detail (i.e., when p « 1). 

This generalization intuitively is appealing when one considers image restoration. One could not 
expect to restore spatial detail that is much finer than the sampling interval, and one ordinarily 
could expect to restore detail that is much coarser from fewer samples. 

(2) The informationally optimized optical design (i.e., trade-off between aliasing and blurring) 
is a function of the SNR. 


1 + 




$^(u,re)|r(w,u;)| 2 * (u,u;) + (K(r L /ay) 2 


dv dw. 


( 6 ) 





Again, this generalization intuitively is appealing when one considers image restoration. In one 
extreme, when the SNR is very low, then the restoration of fine detail is constrained by noise, and 
so it ordinarily would be preferable to avoid substantial blurring (at the cost of aliasing). In the 
other extreme, when the SNR is very high, then the restoration of fine detail is not constrained by 
noise, and so it ordinarily would be preferable to avoid substantial aliasing (at the cost of blurring). 
However, as we will show in Section 5 below, some other constraints may be introduced by the image 
restoration, such as ringing near sharp edges (Gibb’s phenomenon). 
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Figure 6. Variation of information density with image-gathering system design. Results are given for several mean 
spatial details p and SNR’s Aox/rr/y. 
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Figure 7. Variation of information density with image-gathering system design. The informationally optimized design 
is the design for which the image-gathering response designated by p c is selected to maximize the information 
density h g for a given SNR A'ox/er/y. 


As a consequence of the above two generalizations, we limit the following quantitative inves- 
tigations mostly to the mean spatial detail p = l but consider three optical designs (see Fig. 7) 
throughout the remainder of this paper. They are (a) the conventional response p c — 0.7 that is 
also informationally optimum for very low SNR’s, (b) the response p c = 0.45 that is informationally 
optimum for intermediate SNR’s, and (c) the response p c = 0.35 that is informationally optimum 
for very high SNR’s. 
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D. Quantization 


Each discrete signal s g (x,y ) is quantized into k levels for 77 -bit encoding, k = 2T If we divide the 
area A into M by N samples, then the area of A for unit sampling intervals is |v4| = MN. Thus, 
the number of distinguishable states in A is k mn , and the amount of data in A is 

H d = MN log 2 k. 

The associated data density is 


hd = JtN ' 0g2 K = (8) 

It is convenient to let the units of h d be bits even though strictly they are bits per sample. Just 
as the information capacity h c given by Eq. (1) sets a theoretical upper limit on the information 
density h g acquired by image gathering, so the data density h d given by Eq. (8) sets a theoretical 
upper limit on the information density transmitted by digital communication. 

E. Information Efficiency 

Since it ordinarily is desirable to use as few encoding levels as possible, some loss of information 
density is associated with the quantization process. Hence, the information density h q of the 
quantized signal s q (x,y) is closely interrelated with the data density h d . This interrelationship 
suggests the definition of information efficiency as the ratio h q /h d . The units of this ratio are 
bif/bit. This definition of information efficiency is analogous to Khinchin’s definition of “relative 
entropy” as the ratio h/ log m, where h is the entropy of the test, and log m is the maximum value 
of h for the m different symbols of the test. 1 ' Another analogy is Jones’s definition of “information 
efficiency” of a light beam as the information capacity per transmitted photon. 18 

To properly interpret the information efficiency h q /h d , we must account for an important 
difference between continuous and discrete entropies. The data density h d is defined for a discrete 
random variable (i.e. , the quantization levels with a uniform probability density function) for which 
the entropy provides i an absolute measure of randomness. The information density h q is defined for 
a continuous random variable (i.e., the continuous magnitude with a Gaussian probability density 
function) for which the entropy provides a measure of randomness relative to an assumed standard. 
It intuitively is satisfying to adjust the ratio h q jh d so that the theoretical upper limit of information 
efficiency becomes unity. This adjustment occurs when (1) the Wiener spectrum is white 

and band-limited to B, (2) the image-gathering response f g {v, w) is unity within B and zero outside, 
and (3) the quantization intervals are very large compared with the magnitude of the noise. The 
information density h q of the digital data becomes then 


=UI log2 


1 + 


$'r(u,u;)|f- fl (u,U7)| 5 


dv dw. 


( 9 ) 


$' L (v,w)\tg(v,w) 2 * ^ Q 0 (v,w) + (K(T L /<T N ) 2 + k~ 2 


Equation (9) for the information density h q reduces to Eq. (8) for the data density h d when the 
above three conditions are evoked. The final step in this reduction of Eq. (9) to Eq. (8) entails the 
approximation given by 


h g = g lo S2(l + « ) 


log 2 K = V 
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This adjustment of the information density h q leads to a linear encoding of the Gaussian signal 
variation over a dynamic range of —y/3Kcr g to y/3Kcr g , which encompasses 92% of the signal. The 
corresponding quantization interval is A = 2\/3 Ka g /n. Values of s g (x, y) < s g — \/3 cr g are assigned 
the value 0 and values of s g (x, y) > s g -F \/?>(7 a are assigned the value k — 1, where s g is the average 
value of s g (x, y). 

Figures 8 and 9 illustrate the dependence of the information density h q and the information 
efficiency hq/hj on the optical-design parameter p c , the SNR Kai/crj^, and the number of encoding 
levels T], These results suggest the following generalizations: 


(3a) Conventional optical responses (p c = 0.7) limit the information density to h q & 2.2 
bifs. This limit is closely approached when the SNR is Kcr^/a^ k 20 and the number 
of encoding levels is t] & 6 bits. The corresponding maximum information efficiency is 
h q /hd ~ 0.53 bif/bit, but with the reduced information density h q m 1.7 bifs as obtained 
with 3-bit encoding. 







KfJ / (7 K ' 7 l / (7 N K<7 i //(7 k 

(a) p ' 0.7 (b) p 0.45 (c) p^ = 0.35 

Figure 8. Information versus the SNR K ox/cr/v for several encoding levels y. The first row presents the 
information capacity h c and density hq, the second row presents the lost information hq — h c , and the 
third row presents the information efficiency h q /hj. The image-gathering system is characterized by the 
optical-design parameter p c and the SNR AV^/cr/v- The mean spatial detail p = 1. 
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(3b)0ptical responses ( p c = 0.45) that are informationally optimized for intermediate 
SNR’s limit the information density to h q & 3.6 bifs. This limit is closely approached 
when the SNR is 80 and the number of encoding levels is r) « 7. The 

corresponding maximum information efficiency is h q /h ^ ss 0.54 bif/bit, but with 
the reduced information density h q ss 2.7 bifs as obtained with 5-bit encoding. 

(3c) Optical responses ( p c — 0.35) that are informationally optimized for high SNR’s 
limit the information density to h q « 4.7 bifs. This limit is closely approached 
when the SNR is & 240 and the number of encoding levels is r] & 8 . The 

corresponding maximum information efficiency is h q /h j ss 0.55 bif/bit, but with the 
reduced information density hq ss 3.7 bifs as obtained with 7-bit encoding. 

The preferred number of encoding levels for information density, ranging from rj « 6 for 
low SNR’s to r] « 8 for high SNR’s, corresponds closely to those often encountered in practice. 
However, their selection entails some conflict between information density and efficiency. This 
conflict resolves itself with data compression. Furthermore, as these results foreshadow, it is 
the image-gathering system that is designed for highest information density that also provides 
the highest information efficiency with lossless image coding. 
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Figure 9. Information density hq and efficiency hq/ hj versus the encoding level 77 for several SNR’s. The 
image-gathering system is characterized by the optical-design parameter p c and the SNR Ki jl/^N- The 
mean spatial detail p. = 1. 
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4. DATA COMPRESSION 


A. Entropy Versus Information Density 

It is common in the prevailing literature 5 ' 10 to consider information density to be 
synonymous with the entropy used to assess data compression. However, entropy, unlike 
information density, does not distinguish between the properties of the scene that we wish 
to restore and the degradations of the image-gathering process that we wish to minimize. 

For example, whereas aliasing and noise subtract from the information density, these same 
degradations add to the entropy. Since it is not possible to distinguish quantitatively between 
the desired and undesired components of the signal, it also is not possible to measure the 
information density of an image. However, it is possible, at least in theory, to measure its 
entropy. 

The entropy of the digital signal s q (x, y) can be determined as follows. Letp(xi, x 2 , ■■■, x mn) 
be the probability of realizing a particular set of digital data containing MN samples and k 
quantization levels per sample. Then the entropy d q of this data is defined as the logarithm 
of the probable number of alternate, distinguishable sets given by 

j K K K 

9 i = ~m £ E E p(x 1, X 2 , ..., x MN )\og 2 p(xi, * 2 , •••. x Mn)‘ ( 10 ) 

Xl = lX2=l X MN = ^ 


The entropy 0 q given by Eq. (10) is equal to the information density h g given by Eq. (9) only 
if the undesired components of the signal are negligible. However, since these undesired 
components are ordinarily not negligible, the information density h g seldom reaches the 
entropy 0 q (i.e. , h q < 0 q ). 

The amount of computation required to find 0 q given by Eq. (10) is, in practice, prohibitive. 
An upper boundary for the entropy 0 q can readily be found if we assume that each sample is 
independent of its neighbors, i.e., that 

p{x 1, x 2 , ..., X MN ) - p(xi)p(x 2 ) ■ ■ -p(x MN ), 

where X{ — A ki and ki is an integer, 1 < ki < k [Fig. 10(a)]. Letting pi = p(x{) = p(Aki), 
the upper boundary for 0 q is then given by 


°q0 = - Xl p i log 2K' ( U ) 

t=i 

Ordinarily, the values pi are obtained from the probability distribution (histogram) of the 
digital data s q (x, y). If we were to assume that the probability distribution is uniform so that 
all quantization levels are equally likely, then pi = 1 /k and the upper boundary 0 g o of the 
entropy 9 q would be equal to the data density h d , i.e., 

K 1 

9 q 0 = - X] - log 2 « = V = h d . 

AC 


However, neighboring samples ordinarily are not independent. As shown in Fig. 4(a), 
significant correlation exists out to approximately three neighboring samples (in all directions) 
if the mean spatial detail p - 1, and out to about 10 neighboring samples if p = 3. Blurring in 
the image-gathering process will further increase the correlation among neighboring samples 
for the fine spatial detail. In practice, it is common to consider only the nearest samples as 
depicted in Fig. 10. If we consider only the past nearest neighbor [see Fig. 10(b)], then the 
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corresponding entropy 9 q i is defined by 


J K K 

j)\og 2 ( x i,Xj). 


i— 1 j=l 


( 12 ) 



(a) Independent points, 6 p o (b) Past nearest neighbor, 

one dimensional, 6 p1 


(c) Past nearest neighbors, 
two dimensional, 9 p 3 


(d) Nearest neighbors, 
two dimensional, 0 p g 


Figure 10. Samples used to estimate the entropy of the digital data Sq(x,y). 

The entropies 9 q g and 9 q g that include, respectively, the past three nearest samples [Fig. 
10(c)] and the 8 nearest samples [Fig. 10(d)] are defined similarly. Hence, as we include an 
in creasing number of neighbors, we approach the entropy 9 q defined by Eq. (10). 

Figure 11 illustrates the variation of the entropies 9 q q, 9 q \, 9 q 2, and 9 q 3 with the mean 
spatial detail p. The zeroth order entropy 9 q 0 does not account for any of the correlation 
that exists among the neighboring samples. It depends solely on the variance of the incident 
radiance field and on the effects of the image-gathering process (including quantization). 
The magnitude of the higher order entropies decreases as more of the correlation among 
the neighboring samples is accounted for. The independence of the higher entropies from 
the properties of the radiance field and image-gathering system are probably limited to the 
informationally optimized designs considered in this paper. 



Figure 11. Estimates of the entropy 6 q versus the mean spatial detail p. The image-gathering process is 
characterized by the optical design parameter p c , the SNR K<tl/<tn, and the encoding level 77 . 


Another limitation of these results is that the absolute magnitude of the entropies shown 
in Fig. 11 cannot be compared strictly to the information density h q computed by Eq. (9) 
and shown in Figs. 8 and 9. The reason is that the entropies are obtained from a single 
target, such as shown in Fig. 3, in which the rectangles have a fixed orientation, whereas the 
information densities are computed for a circularly symmetric Wiener spectrum derived with 
the assumption that the orientation of these rectangles is random with a uniform distribution. 
Hence, the estimates of information efficiency given below will err on the high side. The reason 
is that the entropy would be higher for the actual radiance field with random edge orientations 
than for the simulated radiance field with a fixed edge orientation. 
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B. Image Coding 


Figure 12 illustrates the lossless data compression method that we use to assess the effect 
of compression on the information efficiency of the transmitted data. The purpose of the 
compression is to translate the string of quantized data s g (x, y) into an encoded string of data 
As e (x,y) that is (usually) a compressed version of s g (x,y). 

Discrete Differential Differential Differential Digital 



Figure 12. Model of lossless predictive compression that uses the past three nearest neighboring samples [as 

depicted in Fig. 10(c)]. 

The predictive compression reduces the redundancy, or correlation, of the string of digital 
data Sq(x,y ) prior to the encoding. The system shown in Fig. 12 is often referred to as a 
third-order predictor because it uses the values of the three past nearest neighbors to predict 
the value that is about to be read out. We select the weighting values ax, a 2 , and 03 so 

that the linear mean-square error estimation E j [s fl (a;, y) — s g (x, y)] 2 j is minimized. This 
minimization, which is referred to as best linear estimate, is commonly favored because of its 
mathematical tractability even though some improvement in performance can often be gained 
when nonlinear functions are used to form the estimate . 9 For the wide-sense stationary input 
radiance field, the correlation between neighboring samples is independent of location and the 
three values a \ , 02 , and 03 can be computed as follows. Let 72(m,n ) be the (normalized) 
correlation between the samples located at ( i — m,j — n ) and (i,j) [see Fig. 10(c)], then the 
desired predictor weighting values are given by the following three simultaneous equations : 9 


'72(0,1)' 


'72(0,0) 72(1,0) 72(1,1)' 

'ax' 

72(1,1) 

= 

72(1,0) 72(0,0) 72(0,1) 

a 2 

72(1,0) 


72(1,1) 72(0,1) 72(0,0) 

. a 3. 


where 72(0, 0) = 1. 

Figure 13 gives the correlation values 72(m, n ) and the predictor weighting values a\, 02 , 
and 03 for three radiance fields and three image-gathering systems. The radiance fields are 
characterized by the mean spatial detail y, and the image-gathering systems are characterized 
by the optical-design parameter p c , the SNR Kay/a^, and the number of encoding levels 
T). The behavior of the correlation and weighting values appeals intuitively. The correlation 
between neighboring samples increases both as the mean spatial detail becomes larger (i.e., 
as y increases) and as it becomes more blurred (i.e., as p c decreases). In the limit, the sum 
of the weighting values (i.e., a\ + 02 + 03 ) approaches unity, which suggests that the new 
sample will be similar in value to the neighboring ones with increasing probability. It also 
is interesting to observe that the correlation 72(0, 1) and 12(1,0) of immediately neighboring 
samples lies between 0.84 and 0.92 when the mean spatial detail is y — 3. This result turns 
out to be in close agreement with the observation made by Gonzalez and Wintz 6 (pg. 298) 
that, in practice, this correlation typically lies between 0.85 and 0.95 for properly sampled 
images. 

The image coding achieves further compression by transmitting the more probable symbols 
in fewer bits than the less probable ones. The Huffman code 5,19 that we use is derived by 





successively merging the two least probable samples of A s g (x,y) into a new sample which 
is assigned a probability equal to the sum of the former two probabilities. This process is 
continued until exhaustion. The result of this process is arranged as a tree that is used to 
determine the code words for the quantized data. 

Figure 14 characterizes the effects of the data compression. The compression h^/h e is 
given by the ratio of data density = T] without coding to data density h e with coding. 
This compression does not vary significantly with either the mean spatial detail or the design 
of the image-gathering system. The compression remains within the range of 1.6 to 1.9, and 
thus approaches the factor of 2 that is often given for lossless data compression. However, the 
information efficiency h q /h e of the encoded data depends significantly on the image-gathering 
system design. These results suggest the following generalization: 

(4) The upper limit of the information efficiency achieved with lossless data compression 
increases from ~ 0.5 to 0.9 bif/bit as the information density of the encoded data 
increases from ~ 2 to 4 bifs. Thus, high information density is transmitted more 
efficiently than low information density. 

This generalization intuitively is appealing since the encoded data contain less image- 
gatherixrg degradation (e.g., aliasing and noise) when the information density is high rather 
than low. 
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Figure 13. Characteristics of the lossless predictive compressor as a function of the mean spatial detail 
fi. Shown are the (normalized) correlation R(m, n) between the neighboring samples and the predictor 
weighting values a. 
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(a) p c = 0.7, Ko l /ct n = 20, 
71=6 


(b) p c = 0.45, Ko L /a N = 80, 
Tl = 7 


(c) p c = 0.35, Kct l /o n = 240, 
ti = 8 


Figure 14. Characteristic of the encoded signal as a function of the mean spatial detail fi. Shown are the 
information density h q , the data compression hj/h e , and the information efficiency hq/hj and h q /h e 
before and after compression, respectively. 

5. IMAGE RESTORATION 
A. Information and Fidelity 

The data-processing algorithm that maximizes the fidelity of the restored image is given 
by the unconstrained Wiener filter 1 " 4 

‘FF w )^g(. v ’ w ) ^ 

$lFFFFFI 2 *J1K w,w ) + {^f) +K ~ 2 

If the radiance-field spectrum <&' L (v,w), the image-gathering response t g (v,w), and the SNR 
Ka L /a N are exactly accounted for in ^(u, w ), then the image fidelity / reaches its maximum 
realizable value f m given by 1 ' 4 


$(t>, w) = 


OO 

= JJ ^l(v, w)r g (v, w)W(v, w)dv dw. 


(14) 


It also is possible, then, to express the Wiener filter ^!(v,w) and the fidelity f m as a function 
of the spectral information density h g (v,w ) as follows : 1,2 


\t(w, w) — 


1 


Tg(v,w) 


l _ 2 -hq(v,w) 


86 









and 


oo 

fm = JJ *' L (v, «0 [l - 2 dv d w 

— oo 

where h q (v, w ) is the integrand of Eq. (9) given by 
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These relationships show that our ability to restore images (restorability) is solely limited by 
the term 2 ~ h ^ v < w ) . 

Since the restorability depends on the spectral information density h q (y,w) rather than 
on the total information density h q , it is not possible to directly ascertain whether increases 
in the information density h q will always increase the restorability. Nevertheless, it seems 
reasonable to expect that the restorability of images ordinarily will be correlated positively 
to the available information density. 


B. Fidelity-Maximized Restorations 

Figure 15 presents fidelity-maximized images for the three informationally optimized 
designs characterized in Fig. 14. The change in the visual quality that occurs with increasing 
information density manifests itself mainly as an increase in the resolution, contrast, and 
clarity. Noise and aliasing artifacts disappear almost entirely as the highest available 
information density is approached. However, ringing near sharp edges now becomes a major 
visual defect. This ringing (Gibbs phenomenon) occurs because of the steep roll-off in the 
Wiener filter. As Schreiber 5 (pg. 92) summarizes: “... it is impossible to have maximum 
sharpness with neither aliasing nor ringing, all at the same time. Since the latter is least 
acceptable, some aliasing and loss of sharpness must be accepted by using a more gradual 
roll-off in the filter.” These results suggest the following important generalization: 

(5) Increases in the information density in the fidelity-maximized images is perceived 
mostly as a decrease in image-gathering degradations (i.e., aliasing artifacts and 
noise). However, defects (i.e., ringing) introduced by the image-restoration process 
become more apparent and may limit the amount of information that is useful for 
image restoration. 
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(a) p c = 0.7, Kct l /ct n = 20, 
TJ= 6; hq =2, h e = 3.2, 
h q /h e = 0.62 


(b) p c = 0.45, Ka L /o N = 80, 
Tl= 7; h q = 3.1, h e = 3.9 , 
h q /h e =0.79 


(c) p c = 0.35, Ka L /a N = 240, 
11 = 8; h q = 3.9, h e = 4.6, 
hq/ he = 0.84 


Figure 15. Images restored with the Wiener filter for three informationally optimized image-gathering 
systems. The systems are characterized by the optical design parameter p c , and SNR Ktr^/api, and 
the encoding level ij. The transmitted data are characterized by information density h q , data density h e , 
and information efficiency h q /h e . The mean spatial detail p = 3. 

Figure 16 illustrates the dependence of the image fidelity f m on the mean spatial detail p for 
the three image-gathering systems characterized by Figs. 14 and 15. As can be seen, the image 
fidelity depends almost solely on the characteristics of the target, even though the resolution, 
sharpness, and clarity of the images restored for maximum fidelity depend perceptibly on the 
available information density. This result suggests the following generalization: 

(6) Image fidelity is not a suitable criterion for assessing the performance of image 
gathering and coding. Not only is it insensitive to the visual flaws of the fidelity- 
maximized images but also to the improvements that are gained in the 
visual quality of these restorations with increasing information density. 

C. Restorations for Visual Quality 


The final step in restoring images for maximum visual quality still must be based on 
perceptual rather than mathematical considerations. For this reason it is necessary to 
introduce some ad hoc modification of the Wiener filter to control adaptively the trade-off 
between the enhancement of spatial detail and the suppression of visual defects. Unfortunately, 
these adaptive controls reduce the quantitative connection between optimum filtering and 
informationally optimized image gathering that we have tried to maintain so far. This seems 
unavoidable as long as visual quality cannot be assessed by some figure of merit. 

The goals of the ad hoc modification of the Wiener filter are to reduce the ringing at sharp 
edges and to enhance the visibility of the fine detail and of the boundaries between areas much 
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larger than those that are barely perceived. To provide these adaptive controls, McCormick 
et al. 4 introduced the Wiener-Gaussian enhancement (WIGE) filter given by 


§i e (v,w) = $(u,u>) {exp -2(7ro- t p) 2 -K(iu7rp) 2 exp -2(na e p) 2 j , 


(15) 


where £ is the enhancement parameter that controls the relative amount of the synthetic- 
high filtered frequency components in the restored image. The standard deviation controls 
the smoothing of the low-pass filtered image, and the standard deviation c e controls the 
smoothing associated with the edge enhancement. Nevertheless, a trade-off remains between 
the enhancement of fine detail and sharp edges and the suppression of ringing. 




Figure 16. Image fidelity f m versus the mean spatial detail p. The image-gathering system is characterized 
by the optical-design parameter p c , the SNR and the encoding level r]. 


The preferred values for the WIGE parameters depend on the target, the design of the 
image-gathering system, and the objectives of the observer. For example, if the image- 
gathering system is informationally optimized for high SNR’s and the target is the random 
radiance field used here as an example, then the trade-off between the suppression of ringing 
and the loss of sharpness in the fidelity-maximized images shown in Fig. 15 is reasonably well 
resolved with cq = 0.4 and cr t - = 0.8. The contrast of the fine detail and sharp edges become 
enhanced increasingly as £ is increased. However, this enhancement is achieved only at the 
cost of general visual quality as well as fidelity. Depending on the objectives of the observer, 
the preferred value for £ ranges typically from 0.2 to 0.8. 4 

Figure 17 presents images restored with the WIGE filter. As above, for the fidelity- 
maximized images shown in Fig. 15, the improvement in image quality with increasing infor- 
mation density is perceived mostly as an increase in clarity. Noise and aliasing artifacts dis- 
appear almost entirely as the highest available information density is approached. Moreover, 
ringing near sharp edges now has been suppressed effectively but at some cost in resolution 
and sharpness. A small overshoot still occurs at the boundaries between areas much larger 
than those that are barely perceived. This overshoot enhances the visibility of the boundaries 
and therefore is often preferred; however, it can be suppressed by reducing the enhancement 
parameter C These results suggest the following generalization: 

(7) The visual quality that can be attained with interactive image restoration improves 
perceptibly as the information density increases to ~ 3 bifs. However, the perceptual 
improvements that can be gained with further increases in information density are 
very subtle. 
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(a) p c = 0.7, K<j l /(T n = 20, 
q= 6; h q = 2, h e = 3.2, 
h q /h e =0.62 


(b) p c = 0.45, Ko l /c n = 80, 
ri= 7; h q = 3.1, h e = 3.9, 
h q /h e = 0.79 


(c) p c =0.35, Ka L / 0 N = 240, 
H = 8; h q = 3.9, h e =4.6, 
hq/hg = 0.84 


Figure 17. linages restored with the WIGE filter for three informationally optimized image-gathering systems. 
The conditions are the same as in Fig. 15. 

The relationship between the available information density and the visual quality of the 
interactively restored images can be expected to depend significantly on the properties of 
the target. Hence, this relationship still must be assessed for a variety of different targets and 
enhancements. 


G. CONCLUDING REMARKS 

The goal of data compression, as it is traditionally stated, is to reduce, as much as possible, 
the number of bits necessary to reconstruct a faithful duplicate of the original picture. The 
effects of data compression are assessed qualitatively by visually comparing the duplicate to 
the original picture. This assessment ignores entirely the degradations that image gathering 
and reconstruction introduce into the original picture. It also ignores the potential capabilities 
of digital processing to reduce the visibility of the degradations caused by image gathering 
and reconstruction and to enhance various features for close scrutiny. 

Our point of view is closer to the one that Schreiber 10 expressed with the question: 
“For a given channel, what relationship between the original scene and the transmitted 
signal produces the ‘best’ picture?” Clearly, this relationship depends on the combined 
performance of image gathering and coding. However, our constraints, aside from the 
communication bandwidth, differ from those of Schreiber. Schreiber was concerned mostly 
with telephotography and televison. These applications are constrained, for commercial 
reasons, mostly by the cost of the image display. By contrast, we are concerned with 
space activities and planetary exploration. These applications, in turn, are constrained 
mostly by the size, weight, and power limitations imposed on the spacecraft instrumentation. 
The complexity of the digital processing required to restore images and enhance features is 
ordinarily not a critical constraint. The latter situation also arises frequently in military 
reconnaissance and medical diagnosis. 

Thus, the goal of this paper has been to develop a method for assessing the combined 
performance of image gathering and coding in terms of the information density and efficiency 
of the transmitted data. This method is based on earlier findings 1,2 that informationally 
optimized image gathering maximizes the fidelity of the images restored by the Wiener filter, 
provided that this filter accounts correctly for the image-gathering degradations. However, an 
important obstacle remains: the fidelity-maximized images exhibit some visual defects such 
as ringing, aliasing artifacts, and noise. 3 Therefore, in practice, it is often desirable to reduce 
these defects with interactive processing. 4 

The method for optimizing the end-to-end performance of image gathering and coding for 
interactive restoration that is suggested in this paper is (1) to assess how much information 



is required to restore and enhance images with sufficiently high visual quality for a particular 

application, and (2) to assess how this information can be acquired and encoded most effi- 
ciently. The preliminary results that we have presented are limited to a single, artificial target. 

However, these results intuitively are attractive and consistent with practical experience. 
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